Systems and Methods for User Interaction with Artificial Reality Environments

ABSTRACT

In one embodiment, a method includes generating a visual interaction tool that moves and extends in a three-dimensional artificial-reality environment according to hand and arm movements of a user. It may be detected that the visual interaction tool intersects a predefined region associated with a virtual item of a first type in the AR environment. The visual interaction tool may attach to the first virtual item. A first operating mode for the visual interaction tool may be selected based on the first type of the first virtual item. The first operating mode may be selected from multiple operating modes for the visual interaction tool. A first input from the user may be received while the visual interaction tool is attached to the first virtual item. First operations with the first virtual item may be performed according to the first operating mode and the first input.

TECHNICAL FIELD

This disclosure generally relates to a user interface for an artificialreality (AR) environment.

BACKGROUND

Artificial reality (AR) is a form of reality that has been adjusted insome manner before presentation to a user, which may include, e.g., avirtual reality, an augmented reality, a mixed reality, a hybridreality, or some combination and/or derivatives thereof. Artificialreality content may include completely generated content or generatedcontent combined with captured content (e.g., real-world photographs).The artificial reality content may include video, audio, hapticfeedback, or some combination thereof, and any of which may be presentedin a single channel or in multiple channels (such as stereo video thatproduces a three-dimensional effect to the viewer). Artificial realitymay be associated with applications, products, accessories, services, orsome combination thereof, that are, e.g., used to create content in anartificial reality and/or used in (e.g., perform activities in) anartificial reality. The artificial reality system that provides theartificial reality content may be implemented on various platforms,including a head-mounted display (HMD) connected to a host computersystem, a standalone HMD, a mobile device or computing system, or anyother hardware platform capable of providing artificial reality contentto one or more viewers.

SUMMARY OF PARTICULAR EMBODIMENTS

The embodiments described in this specification are directed to a visualinteraction tool that improves user interaction within an artificialreality (AR) environment. The visual interaction tool allows users tointeract with a variety of virtual items within an AR environment, suchas user interfaces (UI) in the near field, virtual objects, userinterfaces (UI) in the far field, and surfaces on which the user maymove. However, the disclosed aspects of interaction are exemplary and itis contemplated that the visual interaction tool may be used with anyaspect of interacting with the environment.

Near-field UI interaction is user interaction with UI systems that arein physical proximity to the user within a threshold distance.Near-field UI systems may be right in front of the user in theartificial reality environment (e.g., within arm's reach), such astyping on a keyboard or operating a close-up selection menu. Currentsolutions include a method of virtually mapping a user's hand to allowthe user to directly manipulate the near-field UI. However, it isdifficult to accurately aim and for the system to determine theinteractions the user wishes to perform. Additionally, direct touchcapabilities lack user feedback (e.g., haptic feedback). Finally,requiring the user to perform detailed direct touch motions on anear-field UI increases user fatigue from continually pressing buttons.Thus, there is a need for a solution that is intuitive to a user andincreases accuracy.

Interaction with out-of-reach objects is interaction with objects thatare out of arm's reach. Interaction with out-of-reach objects is aproblem because a user confined to a sphere of movement within an ARenvironment cannot easily interact with or move to objects just beyondtheir range of motion. Current solutions include mapping a virtual handto a position at a distance from the user's hand as a mechanism toincrease the user's reach. However, it is difficult to determine whichof a plurality of objects the user intends to interact with when thereis more than one object in the environment. There is a need for asolution that improves the aiming accuracy of this interaction.

User interaction with far-field UI is interaction with UI systems thatare far away from the user, or have a proximity to the user that isbeyond a threshold distance. Current solutions include laser pointerfunctionality. However, laser pointer solutions have deficient accuracyfor determining which virtual item the user means to interact with, whenthere are multiple virtual items beyond the threshold distance from theuser. Furthermore, when the user needs to make a selection, the motionof pressing a button may cause the laser pointer to deviate from thetarget. There is a need for a solution that improves the aiming accuracyand intuitive function of this interaction.

Moving small distances is enabling the user to make small adjustments totheir location. Current solutions include teleportation, but do notallow the user to make small adjustments to position when precision isrequired. There is a need for a solution that allows the user to makesmall adjustments and that is seamlessly integrated with othercapabilities of user interaction.

Current solutions to the above described interactions are a series ofseparate, distinct tools, each with their own issues. There is a needfor a unifying tool; it is cumbersome for users to have to switchbetween multiple tools when interacting with multiple types of virtualitems. For example, a user may wish to perform different interactions inrapid succession (e.g., grabbing and throwing an object, followed bymoving a short distance, followed by grabbing another object). Toperform these interactions with a series of separate tools requires theuser to first select the appropriate tool for object manipulation,perform the object manipulation interaction, then select the tool formoving short distances, perform the movement interaction, and finallyselect the tool for object manipulation and perform the objectmanipulation. Having to press buttons frequently to select differenttools for every interaction within an AR environment is fatiguing,cumbersome, and inefficient for the user.

The visual interaction tool, in contrast, provides a single tool toallow users to interact with virtual items at any distance and to moveshort distances without cumbersome selection of means for eachcapability. The visual interaction tool comprises a combination ofcapabilities and allows contextually switching between them. The visualinteraction tool, additionally, is designed with intuition of physicalinteraction in mind. Each of the functions of the visual interactiontool may be performed by a user as a close approximation to of thosefunctions as physical interactions. For example, the function of pickingup an out-of-reach object with the visual interaction tool was designedto be a close approximation to the physical motion of picking up anobject at a distance using a tool. With the use of the visualinteraction tool, the function of picking up an out-of-reach object mayfeel like a direct physical interaction with the object even at adistance. As another example, the function of operating a scroll bar ofa user interface with the visual interaction tool was designed to be aclose approximation of the feel of physically interacting with ascroll-bar. In this way, the visual interaction tool may makeinteractions within an AR space feel like intuitive physicalinteractions to a user. In contrast, the existing solutions (e.g., atool based on a laser pointer for selecting) are not closeapproximations of the functionality of physical interaction, which makesthem cumbersome and lacking intuition of use.

The visual interaction tool may be activated in response to receivinginstructions from the user (e.g., hand-tracking, controller-tracking,user arm extension, and user button press). A proximity of the visualinteraction tool may be detected for one or more virtual items. Avirtual item may be selected based on the proximity. An operating modeof the visual interaction tool may be selected based on a type of theselected virtual item. Operations with the visual interaction tool maybe performed with the virtual item according to the operating mode andin response to a user input.

The embodiments disclosed herein are only examples, and the scope ofthis disclosure is not limited to them. Particular embodiments mayinclude all, some, or none of the components, elements, features,functions, operations, or steps of the embodiments disclosed herein.Embodiments according to the invention are in particular disclosed inthe attached claims directed to a method, a storage medium, a system anda computer program product, wherein any feature mentioned in one claimcategory, e.g. method, can be claimed in another claim category, e.g.system, as well. The dependencies or references back in the attachedclaims are chosen for formal reasons only. However, any subject matterresulting from a deliberate reference back to any previous claims (inparticular multiple dependencies) can be claimed as well, so that anycombination of claims and the features thereof are disclosed and can beclaimed regardless of the dependencies chosen in the attached claims.The subject-matter which can be claimed comprises not only thecombinations of features as set out in the attached claims but also anyother combination of features in the claims, wherein each featurementioned in the claims can be combined with any other feature orcombination of other features in the claims. Furthermore, any of theembodiments and features described or depicted herein can be claimed ina separate claim and/or in any combination with any embodiment orfeature described or depicted herein or with any of the features of theattached claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a visual interaction tool extending from a positioncorresponding to the user's hand in an artificial-reality (AR)environment.

FIGS. 2 a, 2 b, and 2 c illustrate an example of using the visualinteraction tool to select a virtual item.

FIGS. 3 a, 3 b, 3 c, and 3 d illustrate an example of using the visualinteraction tool to interact with a user interface (UI) in the nearfield.

FIGS. 4 a, 4 b, 4 c, and 4 d illustrate an example of using the visualinteraction tool to select and manipulate a moveable virtual object.

FIGS. 5 a, 5 b, 5 c, and 5 d illustrate an example of using the visualinteraction tool to move small distances within an artificial-reality(AR) environment.

FIG. 6 illustrates an example of using the visual interaction tool tointeract with a UI in the far field.

FIG. 7 illustrates an example method for interacting with virtual itemsin an artificial-reality (AR) environment.

FIG. 8 illustrates an example computer system.

DESCRIPTION OF EXAMPLE EMBODIMENTS

A visual interaction tool may be activated by a user of an AR system. Inparticular embodiments, the visual interaction tool may be activated inresponse to the user pressing a button. For example, there may be aselection button that prompts the visual interaction tool to beactivated. The visual interaction tool may move and extend in athree-dimensional AR environment according to hand and arm movements ofthe user. When the visual interaction tool is activated, it may operatein an operation mode selected from a plurality of operation modes thatwill be described in further detail below.

FIG. 1 illustrates a visual interaction tool extending from a positioncorresponding to the user's hand in an artificial-reality (AR)environment. The visual interaction tool 102 may extend into thethree-dimensional AR environment 100 from a position corresponding tothe user's hand 104. In particular embodiments, the hand 104 may be thephysical hand of the user. For example, if the user is wearing anaugmented-reality headset that allows the user to see the physicalworld, the user would be able to see his physical hand. In otherembodiments, the hand 104 may be a rendered virtual hand that representsthe user's physical hand. In that case, the rendered virtual hand maymove in accordance with the tracked pose (e.g., position andorientation) of the user's physical hand. Tracking may be achieved byusing computer-vision techniques that track observable features of theuser's physical hands. If the user is holding or wearing a controller,tracking may be achieved by tracking the observable features thecontroller (e.g., IR LEDs on the controller) and/or using theinertial-measuring unit (IMU) of the controller.

Visual interaction tool 102, in particular embodiments, may comprise avisual portion 102 b and an endpoint 102 a. Visual portion 102 b may begenerated based on a ray cast extending from a position corresponding tothe user's hand 104. The direction of the ray cast may depend on thepose of the user's hand 104. The visual interaction tool 102 maycomprise a straight-line segment extending into three-dimensional ARenvironment 100 from a position corresponding to the user's hand 104.Endpoint 102 a may comprise a visual dot. The visual dot may grow invisual weight when it is detected that the visual interaction tool isinteracting with a virtual item. The length of extension of the visualinteraction tool may vary in response to hand and arm movements of theuser. For example, the length of extension of the visual interactiontool may be based on the degree of extension of the user's arm. Forexample, if the user's arm is fully extended, the visual interactiontool may have a long extension, such as 2.5 m. If the user's arm is attheir side, corresponding to zero extension of the user's arm, thevisual interaction tool may have a short extension, such as 70 cm. Inparticular embodiments, the visual interaction tool may extend furtherin proportion to the amount of extension of the user's arm (e.g., thelength of the visual interaction tool may be defined as a function ofthe user's arm extension). In particular embodiments, the length ofextension of the visual interaction tool may be based on the level ofdepression of the selection button. For example, the visual interactiontool may extend longer or shorter depending on whether the selectionbutton is fully depressed or partially depressed.

FIGS. 2 a, 2 b, and 2 c illustrate an example of using the visualinteraction tool to select a virtual item. A virtual item may be aninteractable item in the AR environment, such as an object, userinterface, the ground, etc. Certain virtual items may be associated withone or more predefined regions. A first predefined region may be used todetermine selection of the virtual item. A predefined region maycomprise a volume in a three-dimensional AR environment centered arounda virtual item and/or enclosing the virtual item. The size of apredefined region may be larger than the virtual item. A firstpredefined region may be smaller than a second predefined region. Afirst predefined region may be defined by the volumetric border of thevirtual item itself, or a portion thereof (e.g., in particularembodiments, to select a virtual item, the visual interaction tool mayneed to touch the volumetric border of the virtual item itself, or aselectable portion thereof). A second predefined region associated witha virtual item may be used to determine a bending of the visualinteraction tool towards the virtual item for aiding in selection. InFIG. 2 a, visual interaction tool 202 is a straight-line segment withendpoint 202 a located a distance from virtual item 206. In particularembodiments, the visual interaction tool may bend in an arc towards aselectable virtual item. The bend may be based on a proximity of anendpoint of a visual interaction tool to virtual item. In FIG. 2 b,virtual item 206 is associated with two predefined regions: firstpredefined region 208 and second predefined region 210. First predefinedregion 208 is a sphere centered on virtual item 206. Second predefinedregion 210 is a sphere centered on virtual item 206 and is larger thanfirst predefined region 208. FIG. 2b illustrates an example of a visualinteraction tool bending towards a virtual item based on In FIG. 2 b,endpoint 202 a has touched second predefined region 210, and inresponse, visual interaction tool 202 bends in an arc towards virtualitem 206.

A virtual item may be selected using the visual interaction tool. Thevisual interaction tool may operate in a Selection Mode initially uponbeing activated. The user may not need to press a button to select avirtual item while the visual interaction tool operates in SelectionMode. Selection of a virtual item may be based on a proximity of thevisual interaction tool to the virtual item. Selection of a virtual itemmay occur automatically in response to detecting that the visualinteraction tool intersects a first predefined region associated withthe virtual item. In response to selection of a virtual item, the visualinteraction tool may attach to the virtual item. In FIG. 2 c, virtualitem 206 has been selected based on visual interaction tool 202intersecting first predefined region 208 associated with virtual item206. Thereafter, visual interaction tool 202 is attached to virtual item206. Once endpoint 202 a of visual interaction tool 202 is attached tovirtual item 206, endpoint 202 a may continue to be attached to virtualitem 206 even as visual interaction tool 202 is moved around by theuser's hand 204, causing the virtual item to move along with the visualinteraction tool. This mechanic allows the user to feel as if they arephysically manipulating virtual item 206 using the attached visualinteraction tool 202. Visual interaction tool 202 may detach fromvirtual item 206 when a predefined condition is met (e.g., when the usepresses or releases a button, when visual interaction tool 202 is beyonda certain distance from virtual item 206, when the user gives a voicecommand, when virtual item 206 is moved to a particular predefinedlocation or state, etc.).

Different types of virtual items may be associated with different firstpredefined regions for determining selection. Detecting that the visualinteraction tool intersects the first predefined region associated withthe virtual item (the first predefined region could be the virtual itemitself as well) may be based on a determination that the visualinteraction tool is within a threshold distance from the virtual item.When a virtual item is within a threshold distance from the visualinteraction tool in the AR environment, then it may be detected that thevisual interaction tool intersects the first predefined regionassociated with the virtual item. Selection of a virtual item may bebased on a pointing direction of the visual interaction tool.

A virtual item may correspond to a type of virtual item of a pluralityof types of virtual items. The plurality of types of virtual items couldinclude an object, a user-interface (UI), a scroll bar on a UI, a buttonon a UI, an avatar of another user, or a surface on which a user isconfigured to move. A type of a virtual item may indicate one or moreproperties of the virtual item.

In response to selection of a virtual item, the visual interaction toolmay attach to the virtual item. Based on the type of virtual item thatthe visual interaction tool is attached to, an operating mode for thevisual interaction tool may be selected from a plurality of operatingmodes. The type of the virtual item may indicate that the virtual itemis a manipulatable object. For example, an operating mode for the visualinteraction tool may comprise object manipulation based on the virtualitem being a manipulatable object. Operations with the virtual itemaccording to object manipulation operating mode may be moving thevirtual item in the AR environment. As another example, the type of thevirtual item may indicate that the virtual item is a user interface. Anoperating mode for the visual interaction tool may comprise a userinterface operation based on the virtual item being a user interface. Asyet another example, the type of the virtual item may indicate that thevirtual item is a surface on which the user is configured to move. Anoperating mode for the visual interaction tool may comprise locomotionbased on the virtual item being a surface on which the user isconfigured to move.

In particular embodiments, the type of the virtual item may indicatethat that virtual item is a user interface. The virtual item may bedesigned to be used when it is within a threshold distance from theuser. When the user selects the virtual item of the user-interface type,an operating mode of the visual interaction tool may be selected from aplurality of operating modes. The plurality of operating modes may bepredefined at a system level or custom defined by the application thathas generated the virtual item. The operating mode may comprise a userinterface operating mode selected from a plurality of user interfaceoperating modes, depending on the particular type of user interfaceselected by the visual interaction tool. For example, the plurality ofuser interface operating modes of the visual interaction tool maycomprise a mode for button pressing, a mode for operating a userinterface slide bar, or any other mode according to functionalities ofthe user interface.

FIGS. 3 a, 3 b, 3 c, and 3 d illustrate examples of using the visualinteraction tool to interact with a user-interface (UI) in the nearfield. FIGS. 3a and 3b illustrate an example of the user moving auser-interface (UI) window with an attached visual interaction tool. InFIG. 3 a, visual interaction tool 302 has attached to user-interfacewindow 306. In FIG. 3 b, the user is moving visual interaction tool 302to move user-interface window 306 to another location. Movement of theuser is illustrated by movement of user's hand 304.

In FIG. 3 c, visual interaction tool 302 is attached to scroll bar 308of user-interface window 306. The operations of scroll bar 308 may be toscroll through a plurality of elements of user interface 306. Asillustrated in FIG. 3 c, visual interaction tool 302 may be visiblewhile attached to scroll bar 308. Scroll bar 308 may scroll through theelements of user interface 306 in response to the user moving visualinteraction tool 302. In FIG. 3 c, the elements of user interface 306comprise avatars of other users. This mechanic allows the user to feelas if they are physically manipulating scroll bar 308 using the attachedvisual interaction tool 302.

In FIG. 3 d, button 310 is a virtual item on user interface window 304.In response to determining that button 310 is a button user interface,an operating mode of visual interaction tool 302 may be selected to be abutton pressing mode. In FIG. 3 d, visual interaction tool 302 mayperform button pressing operations comprising pressing button 310according to receiving instructions from the user. For example, visualinteraction tool 302 may remain rigid and would not attach to any of thebuttons when contact is made (e.g., visual interaction tool 302 actslike a stick for pressing buttons). In particular embodiments, inresponse to the visual interaction tool intersecting a first predefinedregion associated with a user interface button (e.g., in this case, theregion may be larger than the button), the visual interaction tool mayautomatically snap to the center of the button to assist the user withaiming. While the visual interaction tool intersects the firstpredefined region associated with the user interface button, the visualinteraction tool may remain snapped to the center of the button.

FIGS. 4 a, 4 b, 4 c, and 4 d illustrate an example of using the visualinteraction tool to select and manipulate a moveable virtual object. Thetype of virtual item 406 may indicate that virtual item 406 is an objectthat can be moved. Selection of object 406 may occur automatically inresponse to the visual interaction tool intersecting a first predefinedregion associated with the object 406. On selecting object 406 withvisual interaction tool 402, visual interaction tool 402 may attach toobject 406 (e.g., when visual interaction tool 402 touches any part ofobject 406, visual interaction tool 402 would appear to attach to object406 without further input from the user). When object 406 is connectedin this way, object 406 may move in the AR environment according to handand arm motions of the user. In FIG. 4 a, visual interaction tool 402 isbeing moved by the user's hand 404 towards object 406. Selection ofobject 406 may occur in response to a determination that visualinteraction tool 402 intersects a first predefined region associatedwith object 406 (e.g., the first predefined region may be any part ofobject 406). In FIG. 4 b, visual interaction tool 402 is shown to beattached to object 406 in response to selection of object 406. In FIG. 4c, object 406 is attached to visual interaction tool 402 and object 406moves in the AR environment along with visual interaction tool 402. Asan example, the user may manipulate the selected object by continuing tohold it at a distance. As another example, the user may perform anaction that may cause visual interaction tool 402 and the attachedobject 406 to retract closer to the user or to retract into a positioncorresponding to the user's hand 404. As another example, the user maymanipulate object 406 using visual interaction tool 402, such as bythrowing object 406. FIG. 4d shows object 406 being held at a distanceby visual interaction tool 402 while the user's hand 404 is movinglaterally to the left, causing the visual interaction tool 402 to bend.Object 406 attached to the end of visual interaction tool 402 would movewith visual interaction action tool 402. During this motion, the usermay decide on a point of release by, for example, pressing or releasinga button. At that instant, object 406 would detach from visualinteraction tool 402. The subsequent motion of object 406 may becomputed according to, for example, the launch speed and angle of object406 at the time of release.

FIGS. 5 a, 5 b, 5 c, and 5 d illustrate an example of using the visualinteraction tool to move small distances within an artificial-reality(AR) environment. An interactable virtual item may be a surface on whichthe user is configured to move (e.g., the floor, a wall, etc.). Anoperating mode for the visual interaction tool may be set to locomotionwhen the tool is attached to such a virtual item. The visual interactiontool may automatically enter the locomotion mode if the virtual itemselected comprises a surface on which the user is configured to move. Asurface on which the user is configured to move may be a floor or wallof the AR environment. When the extended visual interaction tool comesinto contact with a surface on which the user is configured to move, thesurface may be selected by the visual interaction tool. The visualinteraction tool may, on selection of a surface, anchor onto a point onthe surface. In FIG. 5 a, visual interaction tool 502 may extend intothree-dimensional AR environment 500 from a position corresponding tothe user's hand 504. Visual interaction tool 502 is anchored by endpoint502 a onto point 506 on floor 508. Floor 508 is a surface on which theuser is configured to move. The user may use visual interaction tool 502anchored to point 506 on floor 508 to make adjustments to theirlocation. These adjustments may comprise small movements across thetwo-dimensional plane of floor 508. In FIG. 5 a, the user is located ata position when the visual interaction tool 502 is used to attach topoint 506 on floor 508. In FIG. 5 b, the user has made a smalladjustment to their location laterally to the right, resulting in a newlocation that is closer to table 510. The lateral movement may beachieved by the user moving their arm or hand 504 towards the left whilevisual interaction tool 502 is attached to point 506. This motion pushesthe user's position within three-dimensional AR environment 500 towardsthe right, resulting in the user's new location shown in FIG. 5 b. Theamount of the user's movement within AR environment 500 may be directlyproportional to the amount and/or speed of the user's arm or hand 504movement. When the user is satisfied with the new location, the user mayrelease visual interaction tool 502 from floor 508. To continue movingto other locations, the user may repeat the process of attaching visualinteraction tool 502 to floor 508, moving their arm, and releasingvisual interaction tool 502. The mechanics for locomotion provides theuser with fine-grain control over their movements in AR and the feelingthat they are moving through physical means (e.g., pushing and pullingtheir body using a tool anchored to the floor).

FIGS. 5c and 5d illustrate an embodiment where the user may use thevisual interaction tool to push and/or pull them away from/towards theanchor point. In FIG. 5 c, visual interaction tool 502 is anchored byendpoint 502 a onto point 512 on floor 508. Floor 508 is a surface onwhich the user is configured to move. The user may use visualinteraction tool 502 anchored to point 512 on floor 508 to makeadjustments to their location. These adjustments may comprise smallmovements across the two-dimensional plane of floor 508. In FIG. 5 c,the user is located at a position when visual interaction tool 502 isused to attach to point 512 on floor 508. In FIG. 5 d, the user has madea small adjustment to their location by moving backwards, resulting in anew location that is further from desk 514. The movement backwards maybe achieved by the user pressing a button to extend visual interactiontool 502. The extension pushes the user's position away from point 512,backwards in three-dimensional AR environment 500.

In locomotion mode, the user may be configured to move along the surfacein the AR environment in response to instructions from the user. Forexample, the user may move along the surface according to hand and armmotions of the user. As another example, the user could activate avisual interaction tool extending from each of the user's two hands. Theuser may use locomotion mode with both of these visual interaction toolsin an alternating fashion. This mechanic provides the user with thefeeling that they are walking through the three-dimensional AR spacewith physical means (e.g., alternating pushing and/or pulling their bodyusing a tool with each of their hands, simulating the feeling of walkingvia alternating steps).

As another example, the user may be able to move along the surface withmomentum. The user may be able to click and drag within the ARenvironment to give momentum to the user moving along the surface. Asanother example, if the surface is a wall, then the user may beconfigured to move along the wall by climbing. The visual interactiontool may, on selection of the surface, anchor onto a point on the wall.The Locomotion mode may comprise operations allowing the user to climbthe wall.

In particular embodiments, the visual interaction tool may be used tointeract with an interactable virtual item, such as a user interfacethat is positioned beyond a threshold distance from the first user(e.g., beyond a predetermined distance or the arm's reach of the user).Such a user interface may be referred to as a user interface in the farfield.

FIG. 6 illustrates an example of using the visual interaction tool tointeract with a UI in the far field. In FIG. 6, visual interaction tool602 extends from a position corresponding to the user's hand 604 andbends towards virtual item 606 in three-dimensional AR environment 600.Virtual item 606 is a user interface in the far field that is associatedwith another user 610. Virtual items in the far field may be moredifficult for the user to target for selection than other virtual items.To aid the user, there may be a predefined region associated with avirtual item. As illustrated in FIG. 6, a predefined region 608 may be athree-dimensional region or volume in a three-dimensional AR environment600, where the outer border of region 608 is a fixed distance away fromvisual item 606. When it is detected that visual interaction tool 602intersects predefined region 608, visual interaction tool 602 may bendtowards virtual item 606. This mechanic of bending towards selectablevirtual items in the far field not only informs the user of what virtualitems in the far field are selectable, it also decreases the distancebetween the extended visual interaction tool and the virtual item,thereby making selection easier. If the user wishes to make a selection,they may move visual interaction tool 602 towards selectable virtualitem 606 until the two intersect. When that occurs, visual interactiontool 602 may attach to virtual item 606 to provide visual confirmationthat virtual item 606 is currently being selected. While virtual item606 is selected, the user may provide further input to trigger anyfunctionality of virtual item 606. For example, while virtual item 606is selected, the user may press a button to cause the profile data ofthe other user 610 to be shown (e.g., name, profile picture, interests,etc.). In particular embodiments, the user may be given a window of timein which to make a decision about whether to select that virtual item.In that time window, the user may move the visual interaction tool awayfrom the virtual item, preventing selection of the virtual item;otherwise, if the visual interaction tool continues to be within thesecond predefined region, the visual interaction tool may continue toextend towards the virtual item until the two intersect.

A visual aid may alert the user that the user may decide either to movethe visual interaction tool away from the virtual item, or move thevisual interaction tool to intersect the virtual item, completingselection of the virtual item. The visual aid may be that the visualinteraction tool bends in an arc towards the virtual item in response toa detection that the visual interaction tool intersects a secondpredefined region associated with the virtual item. In FIG. 6, visualinteraction tool 101 is intersecting a second predefined regionassociated with virtual item 603, visually shown by the bend in visualinteraction tool 101 towards virtual item 603.

In particular embodiments, the bend in the visual interaction tool maybecome either more pronounced or less pronounced in proportion to thedistance between the visual interaction tool and the virtual item. Abend in response to interaction with a second predefined region may be avisual aid to the user that the visual interaction tool is close tointersecting a visual item and thus close to selection of a visual item.

In particular embodiments, in response to the user moving the visualinteraction tool away from a virtual item, the visual interaction toolmay detect the proximity of an additional virtual item or a plurality ofvirtual items. The visual interaction tool may bend towards anadditional virtual item based on the proximity (e.g., the tool wouldbend towards the closest virtual item).

When detecting that the visual interaction tool intersects the secondpredefined region associated with a virtual item, the visual interactiontool may comprise a straight-line segment and may not intersect thevirtual item. The visual interaction tool may bend in an arc towards thevirtual item in response to detecting that the visual interaction toolintersects a second predefined region associated with a virtual item.The visual interaction tool may comprise a straight-line segment inresponse to detecting that the visual interaction tool no longerintersects the second predefined region of the virtual item.

In particular embodiments, the visual interaction tool may detect theexistence of a far-field virtual item that may be selected and, inresponse, extend automatically to aid the user with selecting that item.For example, when the visual interaction tool is short and not extended,the end point of the tool may be far away from a selectable far-fieldvirtual item. However, had the tool been longer, the tool may besufficiently close to the far-field virtual item to trigger theaforementioned visual aid to help the user select the virtual item. Inparticular embodiments, the visual interaction tool may detect suchscenarios and automatically extend so that its endpoint is closer to thefar-field virtual item. To achieve this, particular embodiments of thevisual interaction tool may comprise a visible first portion generatedbased on a first ray cast extending from a position corresponding to theuser's hand. The first portion of the visual interaction tool may bevisible to the user. The visual interaction tool may further comprise aninvisible second portion that corresponds to a second ray cast extendingfrom the position corresponding to the user's hand. The second portionmay be invisible to the user since it is being used to assess thepossibility of far-field intersections. The second portion may extend adistance greater than the length of the first portion. For example, thelength of the invisible second portion may be 15 meters, 30 meters, orboundless, while the length of the visible first portion may be 0.5, 1,or 1.5 meters. The invisible second portion of the visual interactiontool may be used to detect and select objects that are further away thanthe typical extension of the visible first portion. In response to adetermination that the invisible second portion intersects a first orsecond predefined region associated with a virtual item, the visiblefirst portion of the visual interaction tool may extend to a lengthsufficient for reaching the virtual item in the AR environment.

In particular embodiments, the visual interaction tool may be detectedto be in proximity with each of a plurality of first predefined regionsassociated with a plurality of virtual items. The plurality of virtualitems may comprise a plurality of types. One of the plurality of virtualitems may be selected based on the associated proximities according toone or more priority rules. The priority rules may dictate that thetypes of virtual items may have priority over each other. The priorityrules may dictate that each of a plurality of virtual items has apriority value based on the type of virtual item and further based on aproximity to the user. Of the plurality of virtual items detected to bein proximity to the visual interaction tool, the virtual item with thehighest priority value may be selected. For example, the type of virtualitem being a user interface that is within a threshold distance from theposition corresponding to a user may have the highest priority value.The type of virtual item being an object may have the next-highestpriority value. The type of virtual item being a user interface that isbeyond a threshold distance from the position corresponding to the usermay have the next-highest priority. The type of virtual item being asurface on which the user is configured to move may have the lowestpriority value.

FIG. 7 illustrates an example method for interacting with virtual itemsin an artificial-reality (AR) environment. The method may begin at step710, a visual interaction tool may be generated, wherein the visualinteraction tool moves and extends in a three-dimensionalartificial-reality environment according to hand and arm movements of auser. At step 720, it may be detected that the visual interaction toolintersects a predefined region associated with a first virtual item of afirst type in the artificial reality environment. At step 730, thevisual interaction tool may be attached to the first virtual item. Atstep 740, a first operating mode for the visual interaction tool may beselected based on the first type of the first virtual item, the firstoperating mode being selected from a plurality of operating modes of thevisual interaction tool. At step 750, a first input from the user may bereceived while the visual interaction tool is attached to the firstvirtual item. At step 760, first operations with the first virtual itemmay be performed according to the first operating mode and the firstinput. Particular embodiments may repeat one or more steps of the methodof FIG. 7, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 7 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 7 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates an example method forinteracting with virtual items in an artificial-reality environmentincluding the particular steps of the method of FIG. 7, this disclosurecontemplates any suitable method for interacting with virtual items inan artificial-reality environment including any suitable steps, whichmay include all, some, or none of the steps of the method of FIG. 7,where appropriate. Furthermore, although this disclosure describes andillustrates particular components, devices, or systems carrying outparticular steps of the method of FIG. 7, this disclosure contemplatesany suitable combination of any suitable components, devices, or systemscarrying out any suitable steps of the method of FIG. 7.

FIG. 8 illustrates an example computer system 800. In particularembodiments, one or more computer systems 800 perform one or more stepsof one or more methods described or illustrated herein. In particularembodiments, one or more computer systems 800 provide functionalitydescribed or illustrated herein. In particular embodiments, softwarerunning on one or more computer systems 800 performs one or more stepsof one or more methods described or illustrated herein or providesfunctionality described or illustrated herein. Particular embodimentsinclude one or more portions of one or more computer systems 800.Herein, reference to a computer system may encompass a computing device,and vice versa, where appropriate. Moreover, reference to a computersystem may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems800. This disclosure contemplates computer system 800 taking anysuitable physical form. As example and not by way of limitation,computer system 800 may be an embedded computer system, a system-on-chip(SOC), a single-board computer system (SBC) (such as, for example, acomputer-on-module (COM) or system-on-module (SOM)), a desktop computersystem, a laptop or notebook computer system, an interactive kiosk, amainframe, a mesh of computer systems, a mobile telephone, a personaldigital assistant (PDA), a server, a tablet computer system, anaugmented/virtual reality device, or a combination of two or more ofthese. Where appropriate, computer system 800 may include one or morecomputer systems 800; be unitary or distributed; span multiplelocations; span multiple machines; span multiple data centers; or residein a cloud, which may include one or more cloud components in one ormore networks. Where appropriate, one or more computer systems 800 mayperform without substantial spatial or temporal limitation one or moresteps of one or more methods described or illustrated herein. As anexample and not by way of limitation, one or more computer systems 800may perform in real time or in batch mode one or more steps of one ormore methods described or illustrated herein. One or more computersystems 800 may perform at different times or at different locations oneor more steps of one or more methods described or illustrated herein,where appropriate.

In particular embodiments, computer system 800 includes a processor 802,memory 804, storage 806, an input/output (I/O) interface 808, acommunication interface 810, and a bus 812. Although this disclosuredescribes and illustrates a particular computer system having aparticular number of particular components in a particular arrangement,this disclosure contemplates any suitable computer system having anysuitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor 802 includes hardware for executinginstructions, such as those making up a computer program. As an exampleand not by way of limitation, to execute instructions, processor 802 mayretrieve (or fetch) the instructions from an internal register, aninternal cache, memory 804, or storage 806; decode and execute them; andthen write one or more results to an internal register, an internalcache, memory 804, or storage 806. In particular embodiments, processor802 may include one or more internal caches for data, instructions, oraddresses. This disclosure contemplates processor 802 including anysuitable number of any suitable internal caches, where appropriate. Asan example and not by way of limitation, processor 802 may include oneor more instruction caches, one or more data caches, and one or moretranslation lookaside buffers (TLBs). Instructions in the instructioncaches may be copies of instructions in memory 804 or storage 806, andthe instruction caches may speed up retrieval of those instructions byprocessor 802. Data in the data caches may be copies of data in memory804 or storage 806 for instructions executing at processor 802 tooperate on; the results of previous instructions executed at processor802 for access by subsequent instructions executing at processor 802 orfor writing to memory 804 or storage 806; or other suitable data. Thedata caches may speed up read or write operations by processor 802. TheTLBs may speed up virtual-address translation for processor 802. Inparticular embodiments, processor 802 may include one or more internalregisters for data, instructions, or addresses. This disclosurecontemplates processor 802 including any suitable number of any suitableinternal registers, where appropriate. Where appropriate, processor 802may include one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 802. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 804 includes main memory for storinginstructions for processor 802 to execute or data for processor 802 tooperate on. As an example and not by way of limitation, computer system800 may load instructions from storage 806 or another source (such as,for example, another computer system 800) to memory 804. Processor 802may then load the instructions from memory 804 to an internal registeror internal cache. To execute the instructions, processor 802 mayretrieve the instructions from the internal register or internal cacheand decode them. During or after execution of the instructions,processor 802 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor802 may then write one or more of those results to memory 804. Inparticular embodiments, processor 802 executes only instructions in oneor more internal registers or internal caches or in memory 804 (asopposed to storage 806 or elsewhere) and operates only on data in one ormore internal registers or internal caches or in memory 804 (as opposedto storage 806 or elsewhere). One or more memory buses (which may eachinclude an address bus and a data bus) may couple processor 802 tomemory 804. Bus 812 may include one or more memory buses, as describedbelow. In particular embodiments, one or more memory management units(MMUs) reside between processor 802 and memory 804 and facilitateaccesses to memory 804 requested by processor 802. In particularembodiments, memory 804 includes random access memory (RAM). This RAMmay be volatile memory, where appropriate. Where appropriate, this RAMmay be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, whereappropriate, this RAM may be single-ported or multi-ported RAM. Thisdisclosure contemplates any suitable RAM. Memory 804 may include one ormore memories 804, where appropriate. Although this disclosure describesand illustrates particular memory, this disclosure contemplates anysuitable memory.

In particular embodiments, storage 806 includes mass storage for data orinstructions. As an example and not by way of limitation, storage 806may include a hard disk drive (HDD), a floppy disk drive, flash memory,an optical disc, a magneto-optical disc, magnetic tape, or a UniversalSerial Bus (USB) drive or a combination of two or more of these. Storage806 may include removable or non-removable (or fixed) media, whereappropriate. Storage 806 may be internal or external to computer system800, where appropriate. In particular embodiments, storage 806 isnon-volatile, solid-state memory. In particular embodiments, storage 806includes read-only memory (ROM). Where appropriate, this ROM may bemask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM),electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM),or flash memory or a combination of two or more of these. Thisdisclosure contemplates mass storage 806 taking any suitable physicalform. Storage 806 may include one or more storage control unitsfacilitating communication between processor 802 and storage 806, whereappropriate. Where appropriate, storage 806 may include one or morestorages 806. Although this disclosure describes and illustratesparticular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface 808 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 800 and one or more I/O devices. Computer system800 may include one or more of these I/O devices, where appropriate. Oneor more of these I/O devices may enable communication between a personand computer system 800. As an example and not by way of limitation, anI/O device may include a keyboard, keypad, microphone, monitor, mouse,printer, scanner, speaker, still camera, stylus, tablet, touch screen,trackball, video camera, another suitable I/O device or a combination oftwo or more of these. An I/O device may include one or more sensors.This disclosure contemplates any suitable I/O devices and any suitableI/O interfaces 808 for them. Where appropriate, I/O interface 808 mayinclude one or more device or software drivers enabling processor 802 todrive one or more of these I/O devices. I/O interface 808 may includeone or more I/O interfaces 808, where appropriate. Although thisdisclosure describes and illustrates a particular I/O interface, thisdisclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface 810 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 800 and one or more other computer systems 800 or one ormore networks. As an example and not by way of limitation, communicationinterface 810 may include a network interface controller (NIC) ornetwork adapter for communicating with an Ethernet or other wire-basednetwork or a wireless NIC (WNIC) or wireless adapter for communicatingwith a wireless network, such as a WI-FI network. This disclosurecontemplates any suitable network and any suitable communicationinterface 810 for it. As an example and not by way of limitation,computer system 800 may communicate with an ad hoc network, a personalarea network (PAN), a local area network (LAN), a wide area network(WAN), a metropolitan area network (MAN), or one or more portions of theInternet or a combination of two or more of these. One or more portionsof one or more of these networks may be wired or wireless. As anexample, computer system 800 may communicate with a wireless PAN (WPAN)(such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAXnetwork, a cellular telephone network (such as, for example, a GlobalSystem for Mobile Communications (GSM) network), or other suitablewireless network or a combination of two or more of these. Computersystem 800 may include any suitable communication interface 810 for anyof these networks, where appropriate. Communication interface 810 mayinclude one or more communication interfaces 810, where appropriate.Although this disclosure describes and illustrates a particularcommunication interface, this disclosure contemplates any suitablecommunication interface.

In particular embodiments, bus 812 includes hardware, software, or bothcoupling components of computer system 800 to each other. As an exampleand not by way of limitation, bus 812 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 812may include one or more buses 812, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative. Additionally, although thisdisclosure describes or illustrates particular embodiments as providingparticular advantages, particular embodiments may provide none, some, orall of these advantages.

1. A method comprising, by a computing device: generating a visualinteraction tool, wherein the visual interaction tool (1) moves andextends in a three-dimensional artificial-reality environment accordingto hand and arm motions of a user and (2) is capable of switchingbetween a plurality of operating modes when interacting with differenttypes of virtual items; detecting that the visual interaction toolintersects a predefined region associated with a first virtual item, thepredefined region being a volume larger than and centered around thefirst virtual item; in response to detecting that the visual interactiontool intersects the predefined region, automatically identifying a firsttype of the first virtual item from a plurality of types of virtualitems; selecting a first operating mode for the visual interaction toolbased on the first type of the first virtual item, the first operatingmode being selected from the plurality of operating modes of the visualinteraction tool, wherein the plurality of operating modes correspond tothe plurality of types of virtual items; attaching the visualinteraction tool to the first virtual item; receiving a first input fromthe user while the visual interaction tool is attached to the firstvirtual item; and performing first operations with the first virtualitem according to the first operating mode and the first input.
 2. Themethod of claim 1, further comprising: detaching the visual interactiontool from the first virtual item; attaching the visual interaction toolto a second virtual item of a second type; selecting a second operatingmode for the visual interaction tool based on the second type, thesecond operating mode being selected from the plurality of operatingmodes of the visual interaction tool; receiving a second input from theuser while the visual interaction tool is attached to the second virtualitem; and performing second operations with the second virtual itemaccording to the second operating mode and the second input, wherein thesecond operations are different from the first operations.
 3. The methodof claim 1, wherein the visual interaction tool comprises a visibleportion generated based on a first ray cast from a positioncorresponding to the user's hand, wherein detecting that the visualinteraction tool intersects the predefined region is based on a secondray cast that is invisible to the user, and wherein the second ray castextends a distance greater than a length of the visible portion of thevisual interaction tool.
 4. The method of claim 3, wherein, in responseto the second ray cast intersecting the predefined region associatedwith the first virtual item, the visible portion of the visualinteraction tool extends to the first virtual item.
 5. The method ofclaim 1, wherein detecting that the visual interaction tool intersectsthe predefined region associated with the first virtual item is based ona determination that the first virtual item is within a thresholddistance from the visual interaction tool in the artificial-realityenvironment.
 6. The method of claim 1, further comprising determining apointing direction of the visual interaction tool based on instructionsfrom the user, wherein detecting that the visual interaction toolintersects the predefined region associated with the first virtual itemis based on the pointing direction of the visual interaction tool. 7.The method of claim 1, further comprising: detecting a proximity of thevisual interaction tool to each of a plurality of predefined regionsassociated with a plurality of virtual items of a plurality of types;and selecting one of the plurality of virtual items based on theassociated proximities according to one or more priority rules.
 8. Themethod of claim 1, wherein the visual interaction tool is generated inresponse to receiving instructions indicating that the user has presseda selection button.
 9. The method of claim 8, wherein the visualinteraction tool is detached from the first virtual item in response toreceiving instructions indicating that the user has released theselection button.
 10. The method of claim 1, wherein the first type ofthe first virtual item indicates that the first virtual item is amanipulatable object, and the first operating mode comprises objectmanipulation.
 11. The method of claim 10, wherein the first operationscomprise moving the first virtual item in the artificial-realityenvironment.
 12. The method of claim 1, wherein the first type of thefirst virtual item indicates that the first virtual item is a userinterface, and the first operating mode comprises a user interfaceoperation.
 13. The method of claim 1, wherein the first type of thefirst virtual item indicates that the first virtual item is a virtualrepresentation of another user, and the first operations compriseloading a user profile of the other user.
 14. The method of claim 1,wherein the first type of the first virtual item indicates that thefirst virtual item is a user interface with a slide bar, and the firstoperating mode comprises a slide bar operation.
 15. The method of claim1, wherein the first type of the first virtual item indicates that thefirst virtual item is a surface on which the user is configured to move,and the first operating mode comprises locomotion.
 16. The method ofclaim 15, wherein the first virtual item is a floor or a wall in theartificial-reality environment.
 17. The method of claim 1, wherein whendetecting that the visual interaction tool intersects the predefinedregion associated with the first virtual item, the visual interactiontool is a straight line segment and does not intersect the first virtualitem.
 18. The method of claim 17, further comprising arching the visualinteraction tool towards the first virtual item in response to thedetection that the visual interaction tool intersects the predefinedregion.
 19. One or more computer-readable non-transitory storage mediaembodying software that is operable when executed to: generate a visualinteraction tool, wherein the visual interaction tool (1) moves andextends in a three-dimensional artificial-reality environment accordingto hand and arm motions of a user and (2) is capable of switchingbetween a plurality of operating modes when interacting with differenttypes of virtual items; detect that the visual interaction toolintersects a predefined region associated with a first virtual item, thepredefined region being a volume larger than and centered around thefirst virtual item; in response to detecting that the visual interactiontool intersects the predefined region, automatically identify a firsttype of the first virtual item from a plurality of types of virtualitems; select a first operating mode for the visual interaction toolbased on the first type of the first virtual item, the first operatingmode being selected from the plurality of operating modes of the visualinteraction tool, wherein the plurality of operating modes correspond tothe plurality of types of virtual items; attach the visual interactiontool to the first virtual item; receive a first input from the userwhile the visual interaction tool is attached to the first virtual item;and perform first operations with the first virtual item according tothe first operating mode and the first input.
 20. A system comprising:one or more processors; and one or more computer-readable non-transitorystorage media coupled to one or more processors and comprisinginstructions operable when executed by one or more processors to causethe system to: generate a visual interaction tool, wherein the visualinteraction tool (1) moves and extends in a three-dimensionalartificial-reality environment according to hand and arm motions of auser and (2) is capable of switching between a plurality of operatingmodes when interacting with different types of virtual items; detectthat the visual interaction tool intersects a predefined regionassociated with a first virtual item, the predefined region being avolume larger than and centered around the first virtual item; inresponse to detecting that the visual interaction tool intersects thepredefined region, automatically identify a first type of the firstvirtual item from a plurality of types of virtual items; select a firstoperating mode for the visual interaction tool based on the first typeof the first virtual item, the first operating mode being selected fromthe plurality of operating modes of the visual interaction tool, whereinthe plurality of operating modes correspond to the plurality of types ofvirtual items; attach the visual interaction tool to the first virtualitem; receive a first input from the user while the visual interactiontool is attached to the first virtual item; and perform first operationswith the first virtual item according to the first operating mode andthe first input.